Often a pipeline includes at least one step that depends on the output of
a preceding step. For example, you might
use a step that wants a Python script to
preprocess some data, which must then be used in a subsequent step
to train a model. The output file
dataset config object is a special dataset. It is the dataset that references a location
in a datastore for interim storage of
data and creates a data dependency
between pipeline steps. You can view an output file
dataset config object as an intermediary store
for data that must be passed from a step to
a subsequent step. As advised, you can use the output file dataset config to pass data between steps. In order to do this,
you must define a named output file
dataset config object that references a
location in a datastore. If no explicit
datastore is specified, the default datastore is used. Pass the output file
dataset config object as a script argument in
steps that run scripts, and include code in those
scripts to write to the output file dataset
config argument as an output or read
it as an input. For example, you can
use code that defines an output file
dataset config object for the preprocessed data. In the scripts themselves, you can obtain a reference to the output file dataset
config object from the script argument and use
it like a local folder. You can access the
full version of this code from the reading
at the end of this lesson.